Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle model state in benchmark CI jobs #7432

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Handle model state in benchmark CI jobs #7432

wants to merge 10 commits into from

Conversation

metascroy
Copy link
Contributor

@metascroy metascroy commented Dec 23, 2024 •
edited
Loading

Previously when running a pte file that has model state, our CI would fail with an error like:

2024-12-19 14:15:43.359148-0800 Benchmark[532:16686] [coreml] The input feature for layers_10_attention_SDPA_kv_cache_v_cache must be an MLState, but it was not.

This is because if you build the xcode frameworks with xcode 15, the ExecuTorch CoreML backend is built to not handle MLState variables:

executorch/backends/apple/coreml/runtime/delegate/MLModel_Prewarm.h

Line 11 in fc42a4e

#if !defined(MODEL_STATE_IS_SUPPORTED) && __has_include(<CoreML/MLModel+MLState.h>)

Note the prebuilt frameworks cannot handle MLState (https://github.com/pytorch/executorch/tree/main/extension/benchmark/apple/Benchmark#set-up-the-frameworks).

This PR:

  • Edits the benchmark README.md to mention that Xcode 16.0 must be used to build frameworks that handle pte files that use CoreML MLState
  • Reenables using MLState for iOS18 models
  • Fixes a bug in the build_apple_frameworks.sh related to finding buck

This PR depends on adding ability to specify Xcode version to the macosjob: pytorch/test-infra#6109

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Dec 23, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7432

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 1 Unrelated Failure

As of commit a05b373 with merge base ac7a950 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 23, 2024
@github-actions GitHub Actions
Copy link

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@metascroy metascroy changed the title init Handle model state in benchmark CI jobs Dec 24, 2024
@metascroy metascroy had a problem deploying to upload-benchmark-results December 24, 2024 01:59 — with GitHub Actions Failure
@metascroy metascroy had a problem deploying to upload-benchmark-results December 24, 2024 04:35 — with GitHub Actions Failure
@metascroy metascroy had a problem deploying to upload-benchmark-results December 24, 2024 04:57 — with GitHub Actions Failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guangy10 guangy10 Awaiting requested review from guangy10

@shoumikhin shoumikhin Awaiting requested review from shoumikhin

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@metascroy @facebook-github-bot